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Graphics Calculators in Examinations: A Question of Equity? 



David H. Haimes 



(HAI991 98) 



Analysis of student achievement data following the introduction of graphics calculators to tertiary entrance 
examinations in mathematics in Western Australia suggests the possibility of an equity problem between the 
sexes in their usage. While males out-performed females in a number of categories, the most conclusive 
evidence emerged in questions that were rated as being appropriate for graphics calculator use. 



Background 



For the first time, the use of graphics calculators was allowed in the 1998 Tertiary Entrance Examinations (TEE) in Western 
Australia. Graphics calculators were allowed in the three mathematics subjects, along with those in the physical sciences. The 
decision to allow the use of graphics calculators was based on the rationale that, given the technology exists, then why shouldn’t 
its use be allowed, and that developments with scientific calculators had made it difficult to police their use in examinations 
(personal communication with Curriculum Council officer, November 1999). Some four years’ notice was given for their 
introduction to allow time for schools to make decisions pertaining to curricula and the selection of a suitable calculator. 
Teachers were therefore afforded an opportunity to familiarise themselves with the uses to which the calculators might be put 
and to make the necessary adjustments to their teaching procedures and materials. 

There are three mathematics subjects examined at the TEE level in Western Australia: Applicable Mathematics, Calculus and 
Discrete Mathematics. The former two are considered accessible to the top 30% of the age cohort, while Discrete Mathematics 
is deemed appropriate for the next 30%. Each year, approximately 5000 students sit for Applicable Mathematics, 2000 for 
Calculus and 6000 for Discrete Mathematics. The smaller numbers who study Calculus reflect its status as a pre-requisite for a 
limited number of courses at tertiary level. 



Graphics calculators are a relatively recent technological development and, for many, continue to be an expensive item to 
purchase. Together with indecision as to what stage of schooling their introduction is appropriate, it has meant their utilisation 
as an aid to the teaching of mathematics is yet to become widespread. This is reflected in the somewhat limited scope of the 
research articles that have been published regarding their effectiveness as a tool for instruction and learning in mathematics. 
One area in which a significant amount of research has been published is their use in pre-calculus and calculus classes ; Quesada 
& Maxwell, 1994; . 

Another area that has attracted some attention is sex differences associated with the use of graphics calculators. However, 
studies that have been conducted have produced mixed results. While found that the performance of upper secondary female 
students who used graphics calculators was clearly superior than that of males on items requiring visual-spatial abilities, found 
no apparent differences between males and females in achievement regarding spatial visualisation skills related to graphing linear 
functions. An important difference between Ruthven and Vazquez’s studies is that students in the former had used graphics 
calculators on a regular basis, in the latter study they had not. 

In studies conducted with college students, reported that females experienced more difficulties with the introduction of graphics 
calculators than males. In a subsequent study (1993), they found males were more successful than females on questions that 
required the integration of graphical and algebraic information, but that overall females significantly out-performed males in the 
examination. Furthermore, males were found to be disadvantaged in traditional technique based calculus testing. In a more 
recent study, found that the variable sex did not correlate with Conceptual Mathematical Performance, Basic Algebra Ability, 
Spatial Visualisation, and Mathematical Confidence, and concluded that this supported the equity of graphics calculator-enhanced 
instructional practices in college algebra. 

As a consequence of their meta-analysis of research on the graphics calculator in mathematics education, concluded that the 
use of graphics calculators in examinations was an area of research that was particularly under-represented. This, together with 
the mixed results on sex differences in the use of graphics calculators, suggested a need for further research in these areas. In 
this study, equity between the sexes in their use of the graphics calculator in examinations was the focus, with the following 
questions guiding the collection and analysis of data: 4 
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Following the introduction of graphics calculators in examinations: 

1. Were there differences in achievement between the sexes when questions were categorised by ievei of difficulty, nature 
(procedural or conceptual), or appropriateness for calculator use? 

2. Were there differences in achievement between the sexes according to the curriculum component? 



Data pertaining to student achievement were obtained from the Curriculum Council of Western Australia, the body that 
administers tertiary entrance examinations. These took the form of students’ mean scores for each question (in the case of 
Applicable Mathematics and Calculus) or part question (Discrete Mathematics) on each of the 1997 and 1998 TEE examinations. 
The data were further categorised according to the location of the school (country or metropolitan) and by sex (male or female) 
of the student. 

A second set of data was provided by teachers. For each of the three mathematics courses, a panel of five experienced 
teachers who had taught that course in 1998 were invited to make assessments of the 1997 and 1998 examination papers. With 
respect to each part question, they were asked to rate its difficulty on a scale of 1 to 5, assess whether it was procedural or 
conceptual in nature, and indicate whether the use of a graphics calculator would have been appropriate. Inter-rater reliability 
coefficients were calculated for each mathematics subject and category assessed using the formula 



where MS b is Mean squares between; MS W is Mean squares within and n is the number of cases . 

Data obtained from Teacher Panel members were then aggregated. In the case of level of difficulty, the five ratings were 
summed and each part question was categorised as lower (5- 11), moderate (12- 18), and higher (19- 25) in difficulty. For 
both nature of question and appropriateness of graphics calculator, a majority vote determined the category. Student mean 
scores in both Applicable Mathematics and Calculus were provided for whole, not part, questions with the result that, for some, a 
clear decision was not possible. In these cases, they were categorised as "uncertain." On each of the six examination papers, the 
marks allocated to questions in the various categories were summed, as were the mean scores obtained by students. From 
these, proportions of marks actually obtained were calculated. 

By using the curriculum outline provided by the Curriculum , the author was able to match examination questions to curriculum 
component. All but one question from the six papers was matched in this way. The one exception was a question on the 1997 
Applicable Mathematics examination with parts from both the Descriptive Statistics and Sets, Counting Techniques and 
Probability components. This question was not included in the analysis. 

Differences in proportions according to sex for both 1997 and 1998 were then tested for significance, using: p 1 -p 2 ± 1*960 
lQ 1 (1- P i ) +J2 ^(1-p g)L where p 1 and p 2 are the 



proportions of successes and n 1 and n 2 the corresponding populations. This expression provides 95% confidence intervals; if 
the interval does not include zero, the difference in proportions is significant . 



Table 1 indicates the numbers of students who sat for each of the mathematics examinations in 1997 and 1998. Most of the 
student numbers and proportions remained consistent between the two years, the exception being that the proportion of 
females sitting for the Calculus examination declined further in 1998. 



Table 1 - Numbers (%) of candidates for mathematics examinations 



Methodology 



r = (M S b - 

((MS b - (n-l)MSJ) 



Results 
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Total 


Country 


Metro 


Male 


Female 


Applicable ‘97 


4904 


804 (16) 


4100 (84) 


2612 (53) 


2292 (47) 


Calculus ‘97 


1888 


252 (13) 


1636 (87) 


1270 (67) 


618 (33) 


Discrete '97 


5818 


890 (15) 


4928 (85) 


2301 (40) 


3508 (60) 


Applicable '98 


4976 


832 (17) 


4144 (83) 


2752 (55) 


2224 (45) 


Calculus ‘98 


1885 


253 (13) 


1632 (87) 


1321 (70) 


564 (30) 


Discrete ’98 


5781 


890 (15) 


4891 (85) 


2325 (40) 


3456 (60) 



Inter-rater reliability coefficients for the teacher assessments are reported in Table 2. They were consistently high, indicating a 
strong measure of agreement of responses between the five members of each teacher panel. 

Table 2: Measurement of Agreement Coefficient for Teacher Rating Scales 





Difficulty of 
Question 


Procedural / 
Conceptual 


Appropriate / 
Inappropriate 


1997 Applicable 


.8915 


.7627 


.7953 


1998 Applicable 


.9062 


.6601 


.8976 


1997 Calculus 


.8120 


.6474 


.7059 


1998 Calculus 


.7882 


.6877 


.7339 


1997 Discrete 


.7298 


.7535 


.7180 


1998 Discrete 


.6822 


.7535 


.8851 



The teacher panel assessments are presented in Table 3. Designated are proportions of the total mark that were allocated to 
each of the categories as a result of the teachers' assessments. The ‘uncertain’ category represents questions where a 
clear-cut decision was not able to be obtained. 



Table 3: Teacher Assessment of Examination Items (Percentages) 
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Applicable Calculus Discrete 




1997 


1998 


1997 


1998 


1997 


1998 


Difficulty of 
Question 














Lower 


35 


41 


20 


21 


29 


1 3 


Moderate 


48 


36 


65 


79 


71 


84 


Harder 


1 7 


23 


1 5 


- 


* 


3 


Nature of 
Question 














Procedural 


35 


1 8 


70 


60 


70 


76 


Conceptual 


43 


46 


30 


26 


29 


23 


Uncertain 


22 


36 


* 


1 6 


1 


1 


Graphics 
Calculator Usage 














Appropriate 


22 


23 


20 


1 6 


1 7 


20 


Inappropriate 


61 


64 


75 


74 


83 


78 


Uncertain 


1 7 


1 3 


5 


1 0 


- 


2 



In Table 4, a breakdown of marks according to the respective curriculum component for each of the three mathematics courses 
is presented, it is noticeable that, in Calculus, there were some quite different emphases in the 1998 examination when 
compared with that of 1997. 

Table 4: Proportions of Marks According to Curriculum Component (Percentages) 
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1 997 


1998 


Applicable 


Systems of Equations and Matrices 


22 


26 


Graphs and Solution of Equations 


1 8 


21 


Descriptive Statistics 


1 9 


1 6 


Sets, Counting and Probability 


22 


22 


Random Variables and their Distribution 


1 9 


1 5 






1997 


1998 


Calculus 


Functions and Limits 


1 9 


9 


Theory and Techniques of Calculus 


1 4 


1 6 


Application of Calculus 


40 


46 


Vector Calculus 


1 0 


6 


Complex Numbers 


1 7 


23 






1997 


1998 


Discrete 


Projects, Problem Solving and Investigations 


8 


8 


Data Analysis 


33 


33 


Optimisation 


24 


26 


Growth and Decay 


35 


33 



Differences in proportions by sex according to level of difficulty, nature of question and appropriateness of graphics calculator 
are found in Tables 5, 6 and 7. The results for level of difficulty indicate there were significant differences in achievement 
favouring males in the 1998 examinations in Applicable Mathematics (questions classified as moderate in difficulty) and Discrete 
Mathematics (questions classified as lower and harder in difficulty). Differences in proportions favouring males were significant 
in questions classified as conceptual in nature only in Discrete Mathematics. However, there were significant differences in 
proportions favouring males on questions that were considered appropriate for graphics calculator use in each of the three 1998 
TEE examinations. No such differences in any of the corresponding categories were significant in 1997. 



Table 5: Differences in Proportions by Sex (Male/Female) According to Difficulty of Question 
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1997 


1 998 


Applicable 




Lower 


.743 - .739 


.760 * .754 






(0.029; -0.021) 


(0.030; -0.018) 




Moderate 


.579 - .564 


.524 - .485 






(0.042; -0.013) 


(0.067; 0.012) * 




Harder 


.519 - .493 


.406 - .431 






(0.055; -0.014) 


(0.002; -0.053) 


Calculus 


Lower 


.789 - .793 


.694 - .708 






(0.035; -0.043) 


(0.031; -0.059) 




Moderate 


.718 - .727 


0.551 - 0.543 






(0.034; -0.051) 


(0.057; -0.041) 




Harder 


.449 - .396 


- 






(0.100; 0.005)* 




Discrete 


Lower 


.840 - .829 


.820 - .739 






(0.031; -0.009) 


(0.047; 0.006) * 




Moderate 


.668 - .670 


.627 - .614 






(0.023; -0.026) 


(0.039; -0.012) 




Harder 





.393 - .357 








(0.061; 0.010) * 



Figures in parenthesis are 95% confidence intervals 
* p < 0.05 



Table 6: Differences in Proportions by Sex (Male/Female) According to Nature of Question 





1 997 


1 998 


Applicable 


Procedural 


.705 - .710 
(0.020; -0.031) 


.602 - .598 
(0.031; -0.023) 




Conceptual 


.544 - .548 
(0.043; -0.013) 


.507 - .508 
(0.028; -0.028) 


Calculus 


Procedural 


.739 - .748 
(0.033; -0.051) 


.684 - .671 
(0.059; -0.033) 




Conceptual 


.519 - .483 
(0.084; -0.012) 


.580 - .584 
(0.045; -0.052) 


Discrete 


Procedural 


.772 - .778 
(0.016; -0.028) 


.668 - .656 
(0.037; -0.012) 




Conceptual 


.597 - .581 
(0.042; -0.010) 


.557 - .529 
(0.054; 0.002) * 



Figures in parenthesis are 95% confidence intervals 



http://www.aare.edu.au/99pap/hai99 1 96.htm 

ERIC 7 



BEST COPY AVAILABLE 



Monday, May 21, 2001 



Graphics Calculators in Examinations: A Question of Equity 



Page: 7 



* p < 0.05 



Table 7: Differences in Proportions by Sex (Male/Female) According to Appropriateness of Graphics Calculator 





1997 


1998 


Applicable 


Appropriate 


.693 - .694 
(0.025; -0.027) 


.568 - .515 
(0.081; 0.025) * 




Inappropriate 


.587 - .577 
(0.038; -0.017) 


.548 - .557 
(0.020; -0.036) 


Calculus 


Appropriate 


.745 - .752 
(0.035; -0.049) 


.643 - .562 
(0.130; 0.033)* 




Inappropriate 


.659 - .653 
(0.051; -0.040) 


0.562 - 0.535 
(0.076; -0.022) 


Discrete 


Appropriate 


.798 - .801 
(0.019; -0.023) 


.632 - .605 
(0.052; 0.001) * 




Inappropriate 


.685 - .683 
(0.027; -0.022) 


.645 - .632 
(0.039; -0.011) 



Figures in parenthesis are 95% confidence intervals 
* p < 0.05 



Differences in proportions by sex according to the curriculum component for each of Applicable Mathematics, Calculus and 
Discrete Mathematics are found in Tables 8, 9 and 10, respectively. In the 1998 examinations, there were significant differences 
in Applicable Mathematics in both the Graphs and Solution of Equations and Random Variables and their Distributions components 
favouring males; in Calculus in the Theory and Techniques of Calculus favouring females; and in Discrete Mathematics in the 
Projects, Problem Solving and Investigations component favouring males. Again, no corresponding differences were significant in 
the 1997 examinations. 



Table 8: Differences in Proportions by Sex (Male/Female) According to Curriculum Component - Applicable 

Mathematics 
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1 997 


1 998 


Systems of Equations and 
Matrices 


.711 - .711 
(0.025; -0.025) 


.629 - .617 
(0.039; -0.015) 


Graphs and Solution of 
Equations 


.650 - .627 
(0.051; -0.003) 


.496 - .450 
(0.074; 0.018)* 


Descriptive Statistics 


.621 - .605 


.747 - .744 




(0.034; -0.021) 


(0.027; -0.021) 


Sets, Counting and 
Probability 


.590 - .584 
(0.035; -0.021) 


.489 - .486 
(0.032; -0.024) 


Random Variables and their 
Distributions 


.501 - .463 
(0.066; 0 . 010 )* 


.690 - .650 
(0.067; 0.014)* 


Figures in parenthesis are 95% confidence intervals 




* p < 0.05 






Table 9: Differences 


in Proportions by Sex 


(Male/Female) According 




1 997 


1998 


Functions and Limits 


.721 - .713 


.559 - .561 




(0.051; -0.036) 


(0.047; -0.051) 


Theory and Techniques of 
Calculus 


.797 - .824 
(0.010; -0.064) 


.688 - .735 
(-0.002; -0.091)* 


Application of Calculus 


.637 - .657 


.550 - .525 




(0.061; *0.030) 


(0.074; -0.025) 


Vector Calculus 


.609 - .604 


.581 - .587 




(0.052; -0.042) 


(0.042; -0.055) 


Complex Numbers 


.607 • .623 


.551 • .535 




(0.030; -0.063) 


(0.066; -0.33) 



- Calculus 



Figures in parenthesis are 95% confidence intervals 
• p < 0.05 



Table 10: Differences In Proportions by Sex (Male/Female) According to Curriculum Component - Discrete 

Mathematics 
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1 997 


1 998 


Projects, Problem Solving 


.709 - .695 


.619 - .594 


and Investigations 


(0.038; -0.010) 


(0.051; 0.000)* 


Data Analysis 


.749 - .755 


.621 - .604 




(0.017; -0.029) 


(0.042; -0.009) 


Optimisation 


.727 - .723 


.704 - .701 




(0.029; -0.018) 


(0.027; -0.021) 


Growth and Decay 


.653 - .651 


.596 - .575 




(0.027; -0.023) 


(0.047; -0.005) 



Figures in parenthesis are 95% confidence intervals 
* p < 0.05 



Discussion 

A cursory glance at the results could suggest there is strong evidence that allowing the use of graphics calculators in 
examinations advantages males. However, a more careful examination reveals that such a conclusion may not be altogether 
warranted. 

In examining the data concerning achievement according to the level of difficulty of the examination questions, of those 
categories in which there was a significant difference in proportions, only that of moderate difficulty questions in Applicable 
Mathematics warrants further analysis. Marks allocated to these questions constituted 36% of the total for the paper. 
Furthermore, there was a difference of four percentage points in mean score, favouring males, in the 1998 examination 
compared with 1.5 in 1997. Significant differences found in the Discrete Mathematics examination are not as convincing, as the 
lower difficulty questions represented 13% of the total, while higher difficulty were a mere 3% and there was no corresponding 
category of the latter in 1997. However, with the lower difficulty questions, the difference in mean scores favouring males was 
8 percentage points in 1998, compared with one in 1997. While these results are of interest, it is difficult to draw any definitive 
conclusions as to whether either sex was advantaged according to difficulty of the question. 

Data concerning the nature of the question revealed that males performed significantly better than females on questions 
categorised as conceptual in the Discrete Mathematics examination, these representing 23% of the total marks allocated. The 
difference in mean scores favouring males was three percentage points in 1998 compared with nearly two in 1997. Again, no 
definite conclusion can be drawn about achievement according to the nature of the question. 

In each of the three mathematics courses, there were significant differences favouring males on questions where the use of a 
graphics calculator was considered appropriate. The respective proportions of questions in this category for Applicable 
Mathematics, Calculus and Discrete Mathematics were 23%, 16% and 20%. The differences in mean scores favouring males were 
5, 8 and 3 percentage points in 1998 compared with 1, 0.5 and 0 in 1997, respectively. These results suggest that this group of 
males was more adept than the females at using graphics calculators when answering questions for which their use is considered 
appropriate. This is in line with Boers and Jones’ (1991) findings that females experienced more difficulties than males with the 
introduction of graphics calculators. 

There were four curriculum components in which significant differences in achievement emerged following the introduction of 
graphics calculators. Three of these favoured males, while one favoured females. The component, Random Variables and their 
Distributions in Applicable Mathematics, can be discounted since the difference in proportions was also significant in 1997. The 
difference in mean scores of females and males on the Theory and Techniques of Calculus component, which represented 18% of 
the total marks, was five percentage points in 1998 compared with two in 1997, each favouring females. This confirms Boers 
and Jones’ (1993) finding that males were disadvantaged in traditional technique based calculus testing. 

Significant differences in proportions favouring males were found in the Graphs and Solution of Equations component of 
Applicable Mathematics (representing 21% of the total marks) and Projects, Problem Solving and Investigations of Discrete 
Mathematics (representing 8% of the total marks). In the former, the difference in mean scores favouring males was five 
percentage points in 1998 compared with two in 1997. This lends support to Boers and Jones’ (1993) finding that males were 
more successful than females on questions requiring the integration of graphical and algebraic information. However, little can 
be read into the result for the Projects, Problem Solving and Investigations component as it represented only 8% of the total 
marks, while differences in mean scores favouring males increased from one percentage point in 1997 to three in 1998. 
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Graphics Calculators In Examinations: A Question of Equity 

The results of this study indicate that there may be a problem with equity between females and males in their use of graphics 
calculators in examinations. Before any definite conclusions can be drawn, more research into the issue is necessary. If 
subsequent research confirms the suspicions raised here, then appropriate measures to ensure that females are as comfortable 
and proficient in their use of graphics calculators will need to be implemented. 
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